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VIDEO/TEXT BI-DIRECTIONAL LINKAGE FOR SOFTWARE FAULT 
CLEARANCE APPLICATIONS 

FIELD OF THE INVENTION 

[0001] This invention is directed to a user interface having information screens 
for displaying instructive information. 

BACKGROUND OF THE INVENTION 

[0002] Present and future high capacity printing and reprographic systems 
support high volume document production for commercial and printing center operations, 
with a wide range of input and finishing configurations. Further, operators are provided 
with many job programming options and selections in order to maintain an efficient 
usage of these printing systems. 

[0003] In view of the complexity of these high capacity printing systems, and the 
user interfaces provided for their operation and control, it is to be expected that fault 
conditions will occur in supply unit devices and output imit devices. High capacity 
printing and reprographic systems exist which provide a graphical user interface that 
informs an operator of these faults as they occur and provides instructive information in 
fault screen windows providing the operator with the information necessary to know what 
the fault condition is and also how to correct the fault condition. For example, a typical 
high capacity reprographic system, upon determining the presence of a faidt, will open a 
window on a user interface screen with a text message indicating the source of the fault. 
The system may optionally display a bit mapped graphic image of the section of the 
machine in need of attention. The image may, for example, show a picture of a drawer 
that must be pulled out to replenish a paper supply. 

[0004] While a bit mapped image may be sufficient for simple faults such as an 
out-of-paper condition, it may not be adequate for more complicated faults such as a 
paper jam in the transport mechanism of a reprographic system. Clearing such a fault 
may require multiple steps such as: opening a door, turning a lever clockwise, pulling out 
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a mechanism, lifting a cover, etc. Such a sequence of steps is difficult for an operator to 
follow even when multiple images are provided to illustrate each step because the images 
do not convey information about the movements required to accomplish the task. Arrows 
may be provided to convey this information, however, this requires the operator to 
examine each image and text message careftilly to understand the necessary movements. 

[0005] Therefore, there is a need for a system and method by which an operator 
of a high capacity printing or reprographic system can be given instructions that are 
easily and readily interpreted for clearing fault conditions. Further, it would be 
advantageous if these instructions can be presented in a manner that allows the operator 
to preview the complete procedure, or portions of the procedure, to become familiar with 
the steps involved before performing the procedure. Still further, it would be 
advantageous if instructions could be presented simultaneously in video and text formats 
wherein the text and video are synchronized with each other. 

SUMMARY OF THE INVENTION 

[0006] According to an aspect of the present invention, there is provided a system 
and method for providing information in video art text format via a graphical user 
interface. The system includes a plvirality of video objects, wherein each video object 
corresponds to information such as a respective fault of a reprographic or printing system, 
and wherein each video object further includes a plurality of video segments. A plurality 
of text objects is provided, wherem each text object also provides mformation related to, 
for example, a respective fauh of the reprographic or printing system, and wherein each 
object further includes a plurality of text steps. A bi-directional linkage system Unks 
each of the video segments to one of the text steps and each of the text steps is linked to 
one of the video segments. A video window is provided for displaying the video 
segments, and a text window is provided for displaying one or more of the text steps of a 
selected text object, wherein the video window displays a video segment linked to a 
highlighted text step in the text window and the highlighted text step in the text window 
is linked to the video segment in the video window. 
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DESCRIPTION OF THE DRAWINGS 



[0007] The present invention exists in the construction, arrangement, in 
combination of the various parts of the device, and steps of the method, whereby the 
objects contemplated are attained as hereinafter more fully set forth, specifically pointed 
out in the claims, and illustrated in the accompanying drawings in which: 

[0008] Figure 1 shows an exemplary prior art fault clearance graphical user 
interface screen; 

[0009] Figure 2A shows a fault clearance graphical user interface screen 
illustrating video to text linkage of the present invention; 

[0010] Figure 2B shows a fault clearance graphical user interface screen 
illustrating text to video linkage of the present mvention; 

[00 11] Figure 3 A illustrates an alternate embodiment of the present invention 
showing video instructions presented in a separate window; 

[0012] Figure 3B illustrates an alternate embodiment of the present invention 
showing text instructions presented in a separate window; and 

[0013] Figure 4 provides a flowchart of a particular embodiment of the present 
invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0014] With reference to Figure 1, an exemplary prior art fault window 10 is 
shown as it would appear on a user interface display screen. Fault window 10 provides 
information relevant to faults that occur, in this example, within a unit of a reprographic 
system such as, for example, paper jams, empty supply trays, etc. Text window 12 
indicates that feeder A has a paper jam, where feeder A is a symbolic name of a paper 
feeder supplying the reprographic system. Additional step-by-step instructions relative to 
the fault can be accessed by selecting next button 14 or, alternately, as each step in the 
fault clearance procedure is performed, text window 12 can automatically advanced to 
display text for the next step to be performed. 

[0015] As a visual aid to the operator of the reprographic system, an image 
window 16 is provided shovidng a picture of the unit in need of attention. Typically, the 
image will include an arrow indicating specifically where the fault has occurred or 
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indicating a direction in which a control must be moved. A video aid may also be 
provided, activated by button 1 8, that will play a video of the fault clearance procedvire in 
image window 16. 

[0016] While the fault clearance procedure illustrated in Figure 1 is useful and 
informative, it would be beneficial to provide instructions that are more easily and readily 
interpreted for clearing fault conditions. It would be advantageous if instructions could 
be presented simultaneously in video and text formats wherein the text and video are 
synchronized with each other. 

[0017] Figure 2A shows a preferred embodiment of the present invention that 
provides video and text synchronization by means of a bi-directional linkage. A fault 
correction window 20 is provided on a user interface display that includes a text window 
22, a video window 24 and a video control bar 26. Video control bar 26 provides 
standard video player controls such as, for example, a start button for starting the video, a 
slider for advancing the video in a forward or reverse direction, a rewind to beginning 
button and a forward to end button. Within text window 22, an enumerated list 28 of 
fault clearance steps is provided with a step indicator 30 to indicate the step currently 
being performed. For improved readability, the indicated step is highlighted to provide 
an effective visible cue for the operator and to make text of the relevant step visually 
more distinct. The highlighting may be done by any suitable means such as making the 
text bold, italicizing the text, underlining the text, changing the color of the text, shading 
the remaining inactive steps or other arrangements to provide highlighting. 

[0018] In a preferred embodiment, the steps enumerated in text window 22 are bi- 
directionally linked to the video in video window 24. In a fault clearance step-by-step 
instruction video, when, for example, step 3 is reached as shown, there is a link to text 
window 22 causing the respective step 3 text to be made visually more distinct as 
previously described. To further enhance making the respective step in text window 22 
visually distinct, a step number icon 32 may optionally be displayed in video window 24. 
This creates a strong visual tie between the two different types of information, enabling 
the reader to more quickly sense and compare for faster comprehension. If the video is 
fast forwarded or rewoimd by control bar 26, the text follows along, becoming visually 
accented when its corresponding section of the video is reached. This is accomplished by 
video-to-text linking. 
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[0019] With video-to-text linking, the video and text are linked but separate. That 
is to say, the video and text are separate objects wherein the video object is encoded with 
links to corresponding steps in the text object. In the preferred embodiment, the text is 
not provided withm the video window 24, but instead is located in a scrollable window 
22 near the video window 24. Although provided simultaneously, the video and the text 
can each be viewed separately. An example where this is beneficial is when an operator 
has become familiar with the system on which a fault has occurred and only needs to 
view the steps in text window 22, using the text as a checklist. 

[0020] In addition to video-to-text linking, text-to-video linking is also provided 
in the preferred embodiment. As shown in Figure 2B, an operator has selected step 5 in 
text window 22. A mouse pointer was used in this example, however, any suitable 
selection means, such as a keyboard selection, can be utilized. Selecting step 5 in the text 
window makes that step "hot" and, therefore, the video in video window 24 automatically 
rewinds or fast-forwards to the beginning of step 5 and starts playing the video for that 
step. This is an example of text-to-video linking, however, it also illustrates the bi- 
directional nature of the linking. An operator may select a step in the text window in 
which case the video automatically synchronizes to the text or, on the other hand, the 
operator may use control bar 26 to forward or rewind the video to a desired step in which 
case the corresponding step in the text window becomes highlighted as active. 

[0021] While video window 24 and text window 22 are shown to be 
approximately the same size in Figures 2A and 2B, the respective windows may be 
resized by an operator according to his or her preferences. For example, when an 
operator is primarily concerned with the video for instructive purposes, the operator may 
shrink the text window 22 and expand the video window 24 so the video is more 
discernible from a distance. Similarly, an operator using the text window 22 as a check 
list may wish to shrink the video window 24 and expand the text window 22. Resizing of 
the respective windows can be accomplished by stmdard, well known methods such as 
dragging a side or comer of a window with a mouse pointer to the desired size. 

[0022] Because the video content and text content are derived from separate 
objects as described above, the video may alternately be displayed in a separate fault 
window 34 as shown in Figure 3A while the text is simultaneously displayed in another 
fault window 36 as shown in Figure 3B. Although the video and text are shown in 
separate fault windows and remain separate objects, the above-described bi-directional 
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linkage still exists and the video/text interaction as previously described operates 
substantially unchanged when separate fault windows are employed. For example, an 
operator selecting step 5 in text window 22 within fault window 36 makes that step "hot" 
and, therefore, the video in video window 24 within fauh Avindow 34 automatically 
rewinds or fast-forwards to the beginning of step 5 and starts playing the video for that 
step. 

[0023] Optional resizing of fault windows 34 and 36 provides an operator with 
the ability to enlarge a preferred window or shrink a less desired window as desired. 
Alternately, when separate fault windows 34 and 36 are employed, an operator may close 
one fault window, leaving only the most desired fault window visible on the user 
interface display. 

[0024] The above-described embodunent provides an operator great flexibility 
when viewing fauh clearance instructions. For example, the video in window 24 may be 
rewound and viewed multiple times to gain a familiarity with a new procedure. The text 
in window 22 may be read as steps are automatically highlighted during the ruiming of 
the respective video. A comphcated or confusing step may be selected m window 22 
repeatedly so the video for that step will automatically be replayed each time the 
corresponding step is selected. If the fauh clearance procedure requires activity at some 
distance from the user interface display, the operator may enlarge the most desired 
window, video or text, for easier viewmg while performing the fauh clearance procedure. 
As each step in the fault clearance procedure is completed, the system can be configured 
to automatically advance the video and text to the next step so the operator can complete 
the fauh clearance procedure without further interacting with the user interface. 

[0025] Bi-directional linkage can be implemented in a number of ways. One 
process not requiring alteration of the text or video objects, is to create a separate object 
havmg a table contaming video times for each segment or step. The system software can 
then be configured to automatically advance text window 22 to a corresponding step as 
the video approaches certain time mtervals. On the other hand, if a step is selected in 
window 22 by the operator, the system software can be configured to determine the total 
elapsed tune of prior steps and, from the total elapsed time, calculate a displacement into 
the video object corresponding to the total elapsed time. In this way, the video in 
window 24 can be automatically repositioned to a starting pomt corresponding to the 
selected step in window 22. 
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[0026] Other methods of bi-directional video/text linkage may be alternately 
employed with equal efficacy. Tags may be interspersed within the video object at 
appropriate intervals corresponding to startmg points for each step. These tags can then 
be utilized to automatically highlight the corresponding step in text window 22 as the tags 
are encountered during playback of the video in window 24. When, however, a step in 
text window 22 is selected by the operator, the video may be searched for a tag 
corresponding to the selected step, and the video can then be automatically repositioned 
and replayed from that point. 

[0027] To fiirther clarify bi-directional video/text linkage as utilized in the 
preferred embodiment, a flowchart is provided in Figure 4. The flowchart provided is 
simphfied as much as possible to provide an overview without presenting detail not 
necessary for the understanding of the mvention. For example, the flowchart assumes 
that the video has been provided with embedded tags corresponding to each fauh 
clearance step to be enumerated in text window 22. It fiirther assumes that the operator is 
operating in a "perform" mode wherein the video and text pause at the beginning of each 
step and the system waits for a signal from the fauh device that the previous step has 
actually been completed before continumg automatically. Alternately, in a "view" mode, 
the operator might wish to have the video in window 24 play to completion with the 
corresponding steps in text window 22 scrolling automatically in step with the video. In 
this alternate case, the video would not pause when embedded tags are encountered. 

[0028] Initially, the system software selects video and text objects from a 
database 40, wherein the objects correspond to the particular fauh that has occurred. 
Next, a video window 24 and text window 22 are opened on the user interface display 
xmit with the first video frame displayed in wmdow 24 and the first step highlighted in 
text window 22. At this point, the software pauses 44 waiting for an event to happen. 
The events the flowchart of Figure 4 is concerned with correspond to, in this particular 
example, selection of a video play button, selection of a video stop button, encountering 
an embedded tag during video playback, a hardware signal that a step has been 
completed, a repositioning of the video control, a selection of a specific step in text 
window 22 and a signal that the fault clearance procedure is complete. 

[0029] The detected event is then analyzed and an appropriate action is selected 
for the event 46. In the case of a video play button selection, processing continues at step 
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48 where a video frame pointer is advanced one frame and the subsequent frame is 
displayed m window 24. A determination is then made whether another event has 
occurred 50 and, if not, returns to step 48 for continuous playback of the video segment. 
A new event occurrence will interrupt the playback, and processing will revert back to 
step 46 for selection of an appropriate action. 

[0030] During video playback, a video stop button may be selected in which case 
processing is directed to step 52, however, no action is necessary except to return to step 
44 and wait for another event. Also during video playback, an embedded tag may be 
encountered, and this event is handled at step 54 where text window 22 is advanced to the 
step corresponding to the embedded tag, and the selected step is highUghted. The video 
frame pointer is advanced at step 56 to the first frame of the next step, and that frame is 
displayed in window 24. In the above-described "perform" mode, the system software 
returns to step 44 m order to effect a pause and wait for another event to occur. 

[0031] When a fault clearance step is completed, and a hardware signal causes an 
event to occur, processing proceeds to step 58 where text window 22 is advanced to the 
succeeding step. Following this, the video frame pointer is advanced at step 60 to the 
first frame of the selected step, as identified by an embedded tag immediately preceding 
the corresponding video segment. Processing then returns to step 44 to wait for another 
event. Alternately, processing may continue at step 48 to initiate continuous video 
playback of the selected segment. 

[0032] When a repositioning of the video is selected by operator action on the 
video slider, rewind or fast-forward controls, processing proceeds to step 62. At this 
point, the video frame pointer is repositioned to a corresponding video frame and, at step 
64, the video is searched backward for the corresponding embedded tag to determine a 
corresponding step to be highlighted in window 22. At step 66, the selected video frame 
is displayed in window 24, and the corresponding text is highlighted in window 22. If, 
similarly, the operator selects a step in v^ndow 22, processing proceeds to step 68 where 
the video is searched for an embedded tag corresponding to the selected step. When 
found, the video frame pointer is repositioned to that point and processing then continues 
at step 66 for updating both of the video and text windows. 



[0033] When the fauh clearance procedure is completed, an event is triggered, 
and processing is directed to step 70 where the video and text windows are closed. The 
system then reverts to a normal processing status. 

[0034] System software programs configured for the above-described 
embodiments may be developed in a number of high level languages. For example, the 
software may be programmed directly in the Java language offered Sun Microsystems, 
Inc. Alternately, a Java development environment may be employed to simplify Java 
programming. Two examples readily available are Visual Cafe from Symantec 
Corporation and JBuilder from Borland Software Corporation. Skilled programmers may 
also prefer to program embodiments directly with either the C or the C++ languages. 
Other programming languages are also available that provide the necessary functionality 
to program embodiments of the present invention. 

[0035] The invention has been described with reference to the preferred 
embodiments. Obviously, modifications and alterations vnll occur to others upon a 
reading and understanding of this specification. It is intended to include all such 
modifications and alterations insofar as they come within the scope of the appended 
claims or the equivalents thereof. 
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